Skip to content

feat(table): WKB encoding + GeoArrow conversion for geometry/geography - #1138

Merged
laskoviymishka merged 10 commits into
apache:mainfrom
happydave1:feat/wkb
Jun 18, 2026
Merged

feat(table): WKB encoding + GeoArrow conversion for geometry/geography#1138
laskoviymishka merged 10 commits into
apache:mainfrom
happydave1:feat/wkb

Conversation

@happydave1

@happydave1 happydave1 commented May 28, 2026

Copy link
Copy Markdown
Contributor

Fixes #991.

Added metadata support in arrow_utils.go, created a WKTToWKB helper in table/internal/geo_codec.go which uses go-geom to convert WKT to WKB, and added a round trip Arrow to Parquet test which tests if geoarrow extension metadata survives. Also tests Iceberg schema to Arrow schema round trips.

@happydave1
happydave1 marked this pull request as ready for review May 28, 2026 19:26
@happydave1
happydave1 requested a review from zeroshade as a code owner May 28, 2026 19:26

@zeroshade zeroshade left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In general this looks good to me, though I'm not AS familiar with the geo stuff here.

@dwilson1988 or @paleolimbot would either of you be able to take a quick look here for verification?

@paleolimbot paleolimbot left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool!

The test cases from apache/arrow-rs#10065 are my most recent attempt at a succinct list of the special cases to take care of, although I've tried to note them inline here, too. The difference between Parquet and Iceberg for CRSes is that Iceberg requires authority:code (or PROJJSON that is offloaded into a table property). You can error for that case (I left inline suggestions about how to convert from PROJJSON to authority:code).

Comment thread table/arrow_utils.go Outdated
Comment on lines +1845 to +1848
func icebergCRSToGeoArrowMetadata(crs string) geoarrow.Metadata {
if crs == "OGC:CRS84" {
return geoarrow.NewMetadata()
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is reversed I think...if the iceberg CRS is "", the GeoArrow CRS is "OGC:CRS84" (unless NewMetadata().

Comment thread table/arrow_utils.go Outdated
Comment on lines +1850 to +1858
if strings.HasPrefix(strings.ToLower(crs), "srid:") {
id := crs[len("srid:"):]
raw, _ := json.Marshal(id)

return geoarrow.Metadata{
CRS: raw,
CRSType: geoarrow.CRSTypeSRID,
}
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is one special case here: "srid:0" maps to an omitted GeoArrow CRS.

Comment thread table/arrow_utils.go Outdated
Comment on lines +1859 to +1864
raw, _ := json.Marshal(crs)

return geoarrow.Metadata{
CRS: raw,
CRSType: geoarrow.CRSTypeAuthorityCode,
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe there is one special case here: if the Iceberg CRS is projjson:<table property>, the CRS is the JSON object in that table property field. You should probably either support that or error with a note saying that it's not supported.

Comment thread table/arrow_utils.go Outdated
Comment on lines +1808 to +1810
if len(meta.CRS) == 0 && meta.CRSType == "" {
return "OGC:CRS84", nil
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the GeoArrow CRS is omitted from the extension metadata, the Parquet equivalent is "srid:0". If the GeoArrow CRS is "OGC:CRS84" or "EPSG:4326", the canonical Parquet CRS is "omitted" (i.e., default). I believe the Parquet and Iceberg CRS definitions are in sync now but it's worth double checking.

Comment thread table/arrow_utils.go Outdated
Comment on lines +1819 to +1826
switch meta.CRSType {
case geoarrow.CRSTypeSRID:
return "srid:" + crs, nil
case geoarrow.CRSTypeAuthorityCode:
return crs, nil
default:
return "", fmt.Errorf("unsupported geoarrow CRS type %q", meta.CRSType)
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the CRSType is PROJJSON, this is a important case to handle (because most GeoArrow producers produce that). You can convert most of these to authority:code by looking for the id member that looks like this "crs": {..., "id":{"authority": "OGC", "code": "CRS84"} or this "crs": {..., "id":{"authority": "EPSG", "code": 3857}. It would also be a good idea to convert the two "lonlat" CRSes ("EPSG:4326" and "OGC:CRS84") to the Iceberg "default" canonically.

When you can't extract an authority:code, this would need to be written to a table property and written to the CRS field as (projjson:<the table property name>). Probably easier to error for that case for now.

@dwilson1988

Copy link
Copy Markdown
Contributor

In general this looks good to me, though I'm not AS familiar with the geo stuff here.

@dwilson1988 or @paleolimbot would either of you be able to take a quick look here for verification?

Happy to re-review if needed, but looks like @paleolimbot is already on it! Ping me if you want a second set of eyes.

@happydave1

happydave1 commented Jun 9, 2026

Copy link
Copy Markdown
Contributor Author

Thank you @paleolimbot and @zeroshade for reviewing.

I believe I have addressed each comment by:

  1. updating icebergCRSToGeoArrowMetadata to handle the default CRS case ("OGC:CRS84") as expected.
  2. updating icebergCRSToGeoArrowMetadata to handle case "srid:0"
  3. erroring out the projjson case in both icebergCRSToGeoArrowMetadata and geoArrowCRSToIcebergCRS - this can be addressed in a separate PR
  4. updating TestIcebergGeoTypesToArrowSchema to mirror tests found in Add tests and fix corner cases for Parquet/GeoArrow extension type conversion arrow-rs#10065 (omitted test cases with partial projjson)
  5. mapping geoarrow crs of "OGC:CRS84" and "EPSG:4326" to the default CRS "OGC:CRS84" so that iceberg.GeographyTypeOf and iceberg.GeometryTypeOf both create geo types with omitted ("") crs.

I would appreciate a second pass whenever you guys have a chance, thanks!

@laskoviymishka laskoviymishka left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work threading the WKB / GeoArrow metadata through the schema conversion. The SRID encoding, the planar vs spherical edge handling, and the round-trip tests are all looking solid.

There are two things I’d want fixed before merge.

First, the EPSG:4326 handling looks asymmetric. geoArrowCRSToIcebergCRS collapses EPSG:4326 to OGC:CRS84 on read, but icebergCRSToGeoArrowMetadata writes EPSG:4326 back out as-is. Those two are not interchangeable because the axis order is different. That means a geometry("EPSG:4326") field — which is what PyIceberg emits — silently comes back as geometry("OGC:CRS84"). So the round trip is not actually equal. The existing CRS test uses EPSG:4267, so it misses this branch. I think we should either make the conversion symmetric or stop collapsing it, and add a regression test specifically for EPSG:4326.

Second, the projjson behavior is also asymmetric. The read path returns a clean error, but the write path panics. That panic propagates through VisitGeometry / VisitGeography with no recovery, so converting a table with a projjson CRS can crash the reader. This should return an error instead.

I left a few smaller comments inline as well: the authority_code / wkt2:2019 pass-through, discarded json.Marshal errors, and a couple of nlreturn gaps that will likely fail CI lint.

Once the two main issues above are fixed, I’m happy to take another pass and approve.

Comment thread table/arrow_utils.go Outdated
}
}

if crs == "OGC:CRS84" || crs == "EPSG:4326" {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

EPSG:4326 and OGC:CRS84 aren't the same CRS — they have opposite axis order (CRS84 is lon/lat, EPSG:4326 is lat/lon). Collapsing them here means a geometry("EPSG:4326") field — which is exactly what PyIceberg emits via ga.wkb().with_crs("EPSG:4326") — reads back as geometry("OGC:CRS84"), so the schema silently changes on the read side.

The write path doesn't share the collapse either: icebergCRSToGeoArrowMetadata emits EPSG:4326 verbatim, so GeometryType{crs:"EPSG:4326"} → Arrow → GeometryType{crs:"OGC:CRS84"} and the two aren't Equals.

I'd drop the EPSG:4326 case here and treat it as a distinct authority code. If we genuinely want them unified, it has to be symmetric — normalize on both the write and read sides so the round trip is stable — and it shouldn't live silently at the schema layer. Either way a round-trip regression test for EPSG:4326 specifically would lock it down. wdyt?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

EPSG:4326 and OGC:CRS84 aren't the same CRS

They are for the purposes of the GeoArrow and Parquet specifications, which explicitly define the axis order for these cases

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe that a round trip regression test is definitely in order for this behavior.

@paleolimbot, is the asymmetric behavior here expected or should we collapse "EPSG:4326" at the write level too? (i.e. add a conditional in icebergCRSToGeoArrowMetadata)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The ideal behaviour is that all of these become the Iceberg default CRS:

  • "crs": "EPSG:4326" and "crs": "OGC:CRS84" (case insensitive)
  • "crs": {..., "id":{"authority": "OGC", "code": "CRS84"}
  • "crs": {..., "id":{"authority": "EPSG", "code": 4326}
    • "crs": {..., "id":{"authority": "EPSG", "code": "4326"}

When reading an iceberg default CRS to GeoArrow, emit "crs": "OGC:CRS84". I think your PR is currently missing this case.

This asymmetry is helpful to improve the compatibility of iceberg tables (e.g., for readers that can't or don't want to understand CRSes and only handle the default case).

Comment thread table/arrow_utils.go Outdated
}
}

if strings.HasPrefix(strings.ToLower(crs), "projjson:") {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This panics on projjson, but the symmetric read function geoArrowCRSToIcebergCRS returns a clean error for the same case. The asymmetry is the problem: VisitGeometry/VisitGeography have no error return and nothing recovers, so a GeometryType with a projjson: CRS detonates the whole conversion at TypeToArrowType rather than surfacing an error.

I'd return an error here and thread it through the same way the read path does, so a projjson CRS fails gracefully instead of crashing the process.

Comment thread table/arrow_utils.go Outdated

iceType, err := geoArrowMetadataToIcebergType(wkb.Metadata())
if err != nil {
panic(fmt.Errorf("%w: %v", iceberg.ErrInvalidSchema, err))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The %v on err drops it out of the wrapped chain, so errors.Is/errors.As against the underlying error won't work downstream. I'd use %w for both:

panic(fmt.Errorf("%w: converting geoarrow metadata: %w", iceberg.ErrInvalidSchema, err))

Comment thread table/arrow_utils.go
return "srid:" + crs, nil
case geoarrow.CRSTypePROJJSON:
return "", errors.New("geoarrow CRS type projjson not supported yet")
default:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CRSTypeAuthorityCode and CRSTypeWKT22019 both fall into default and get returned as-is. For authority_code that's almost right but loses the type tag; for WKT2:2019 it hands a multi-KB WKT2 blob straight back as the Iceberg CRS string, which isn't a valid CRS identifier.

I'd add an explicit CRSTypeAuthorityCode case, and for CRSTypeWKT22019 either error like projjson does or reduce it to the authority code — whichever we pick, a test for each so the behavior is pinned.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The CRSType, if it's coming from JSON, is just a hint, is not required, and is often absent. Here you probably want to:

  • Check if crs is a JSON object. If it is, check the id member and paste together crs["id"]["authority"], :, and crs["id"]["code"]. If any of those are missing, error for an unsupported CRS>
  • Check if crs is a string. If it's shorter than 32 characters, let it through verbatim. There's no official restriction on allowed characters in authorities or codes but the length check should reject anything questionable.

Comment thread table/arrow_utils.go Outdated

raw, _ := json.Marshal(crs)

return geoarrow.Metadata{

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For an authority-code CRS like EPSG:4326 this emits {"crs":"EPSG:4326"} with no crs_type, but the GeoArrow spec wants crs_type: "authority_code" for AUTHORITY:CODE strings. Without it the encoding is ambiguous, and combined with the collapse in geoArrowCRSToIcebergCRS it's the other half of the EPSG:4326 round-trip break.

I'd set CRSType: geoarrow.CRSTypeAuthorityCode when the CRS matches the authority:code shape, so the write side is unambiguous and pairs cleanly with an explicit authority_code read case.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"crs_type": "authority_code" is optional and no consumer actually requires this, but it is polite to include it (I put this in the Arrow C++ export path).

Comment thread table/arrow_utils.go Outdated
return geoarrow.NewMetadata() // srid:0 maps to omitted GeoArrow CRS
}

raw, _ := json.Marshal(id)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

errcheck is enabled outside _test.go, so the discarded errors here and at the json.Marshal(crs) below will fail CI. Marshaling a string can't actually fail, but the linter doesn't know that — I'd either build the raw JSON without the error return (e.g. json.RawMessage(strconv.AppendQuote(nil, id))) or add a //nolint with a one-line why.

Comment thread table/arrow_utils.go

func geoArrowCRSToIcebergCRS(meta geoarrow.Metadata) (string, error) {
if len(meta.CRS) == 0 && meta.CRSType == "" {
return "srid:0", nil

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A bare ga.wkb() field with no CRS — the PyIceberg default and what older clients emit — reads back as geometry("srid:0") rather than geometry(), which isn't Equals to a default-CRS geometry.

Separately, an empty meta.CRS with a non-empty CRSType skips this early return and falls into the switch, so CRSTypeSRID yields "srid:" with an empty id and GeometryTypeOf accepts it silently. I'd return the OGC:CRS84 default for the bare case and guard the empty-CRS-with-CRSType combination, with a test for bare geoarrow.wkb.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bare ga.wkb() field with no CRS — the PyIceberg default and what older clients emit — reads back as geometry("srid:0") rather than geometry(), which isn't Equals to a default-CRS geometry.

This is the correct behaviour: the GeoArrow default does not equal the Iceberg default. PyIceberg is probably wrong here.

Separately, an empty meta.CRS with a non-empty CRSType skips this early return

This should be fixed...the CRSType can actually just be ignored for the purposes of this function (it's purely a hint)

Comment thread table/arrow_utils.go
}
}

func icebergCRSToGeoArrowMetadata(crs string) geoarrow.Metadata {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small thing while we're here: strings.ToLower(crs) runs twice (here and for the projjson check) and the slice crs[len("srid:"):] indexes the original-case string after a lowercased prefix check. I'd hoist lower := strings.ToLower(crs) once and reuse it. Also worth noting the EPSG:4326 match in the read function is case-sensitive, so epsg:4326 falls through — strings.EqualFold would close that.

Comment thread table/arrow_utils.go

@paleolimbot paleolimbot left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few more details but this is looking good!

Comment thread table/arrow_utils.go Outdated
}
}

if crs == "OGC:CRS84" || crs == "EPSG:4326" {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

EPSG:4326 and OGC:CRS84 aren't the same CRS

They are for the purposes of the GeoArrow and Parquet specifications, which explicitly define the axis order for these cases

Comment thread table/arrow_utils.go

func geoArrowCRSToIcebergCRS(meta geoarrow.Metadata) (string, error) {
if len(meta.CRS) == 0 && meta.CRSType == "" {
return "srid:0", nil

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bare ga.wkb() field with no CRS — the PyIceberg default and what older clients emit — reads back as geometry("srid:0") rather than geometry(), which isn't Equals to a default-CRS geometry.

This is the correct behaviour: the GeoArrow default does not equal the Iceberg default. PyIceberg is probably wrong here.

Separately, an empty meta.CRS with a non-empty CRSType skips this early return

This should be fixed...the CRSType can actually just be ignored for the purposes of this function (it's purely a hint)

Comment thread table/arrow_utils.go
return "srid:" + crs, nil
case geoarrow.CRSTypePROJJSON:
return "", errors.New("geoarrow CRS type projjson not supported yet")
default:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The CRSType, if it's coming from JSON, is just a hint, is not required, and is often absent. Here you probably want to:

  • Check if crs is a JSON object. If it is, check the id member and paste together crs["id"]["authority"], :, and crs["id"]["code"]. If any of those are missing, error for an unsupported CRS>
  • Check if crs is a string. If it's shorter than 32 characters, let it through verbatim. There's no official restriction on allowed characters in authorities or codes but the length check should reject anything questionable.

Comment thread table/arrow_utils.go Outdated

raw, _ := json.Marshal(crs)

return geoarrow.Metadata{

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"crs_type": "authority_code" is optional and no consumer actually requires this, but it is polite to include it (I put this in the Arrow C++ export path).

Comment thread table/arrow_utils_test.go
@happydave1
happydave1 force-pushed the feat/wkb branch 2 times, most recently from 03ba187 to e4fbac6 Compare June 10, 2026 20:54
@happydave1

Copy link
Copy Markdown
Contributor Author

Hi @laskoviymishka @paleolimbot, thank you guys for giving my PR a second look. I appreciate the comments!

I believe I have addressed all comments but feel free to correct me if I missed anything!

  1. Regarding EPSG:4326 behavior, I have added a round trip test specifically for this CRS, pinning its behavior. I have collapsed the write behavior for EPSG:4326 so that it behaves consistently across reads and writes.

  2. Added read side (arrow field to Iceberg type) tests ensuring that partial projjson CRS's

  • "crs": "EPSG:4326" and "crs": "OGC:CRS84" (case insensitive)
  • "crs": {..., "id":{"authority": "OGC", "code": "CRS84"}
  • "crs": {..., "id":{"authority": "EPSG", "code": 4326}
  • "crs": {..., "id":{"authority": "EPSG", "code": "4326"}

all map to default Iceberg CRS.

  1. Added a test confirming that the projjson write case cleanly errors from the public entrypoint (table.TypeToArrowType)

  2. Added read side projjson support, write side cleanly errors.

  3. Updated error wrapping to use %w instead of %v

  4. Added an error case for wkt2:2019 on the read side. Also added a test which pins this behavior. Note that this relies on crs_type being set to "wkt2:2019". In the case that crs_type is not set, the write path will go through a length check which it will likely fail since wkt2:2019 crs type is quite verbose. We can work on supporting the crs_type not set path in a future PR.

  5. Added "crs_type": "authority_code" for crs with AUTHORITY:CODE form.

  6. Added //nolint:errcheck for discarded JSON marshalling errors involving strings.

  7. Added a test case for a geoarrow metadata with only "crs_type" set.

  8. Documented best effort edge hint.

  9. Hoisted strings.ToLower(crs) in write case and used strings.EqualFold` to check equality in read case. Added tests to guard this.

@paleolimbot paleolimbot left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for this! A few details but this is looking great from the GeoArrow metadata side (with apologies if I've misunderstood any of the Go here)

Comment thread table/arrow_utils.go
Comment thread table/arrow_utils.go Outdated
Comment thread table/arrow_utils.go Outdated
Comment thread table/arrow_utils_test.go Outdated
Comment thread table/arrow_utils_test.go Outdated
Comment thread table/arrow_utils_test.go
Comment thread table/arrow_utils_test.go

@laskoviymishka laskoviymishka left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work on this round, most of the earlier feedback is in now:

  • EPSG:4326 ↔ OGC:CRS84 collapse is symmetric
  • authorityCodeCRS regex is anchored
  • CRSTypePROJJSON hard-error branch is gone
  • test split / naming looks cleaner
  • %w double-wrap, errcheck, planar-geography docs, and EdgePlanarGeometry are cleaned up

I’d hold before merge, mostly for one real bug.

The SRID write path slices the identifier out of the lowercased CRS string, so non-numeric SRIDs get case-folded. srid:MyDB round-trips as srid:mydb. Numeric SRIDs are fine, which is probably why this stayed hidden. Slicing from the original crs should fix it. I’d also add a srid:4326 case in typeCases to cover the common path.

One other thing: go-geom is now a production dependency for test-only code. I’d lean toward moving WKTToWKB into a test package, but that’s not blocking.

The rest is cleanup: dead len(meta.CRS) guards, EqualFold on an already-lowercased string, unreachable codeNum.String() branch, checkCRSSJSON typo, and an invalid-WKT negative test.

One aside: the projjson write path still panics rather than returning an error. Fine if it only goes through the visitor recover path, but any direct caller would still crash.

Once those are in, happy to take another pass.

Comment thread table/arrow_utils.go Outdated
func icebergCRSToGeoArrowMetadata(crs string) geoarrow.Metadata {
lowerCRS := strings.ToLower(crs)
if strings.HasPrefix(lowerCRS, "srid:") {
id := lowerCRS[len("srid:"):]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The id is sliced out of the lowercased string, so a non-numeric SRID gets silently case-folded: srid:MyDB goes in, mydb gets marshalled, and it round-trips back as srid:mydb. Numeric SRIDs are fine, which is why it's latent.

I'd slice the original crs instead — id := crs[len("srid:"):] — and keep the HasPrefix check on lowerCRS.

The iceberg_to_arrow_round_trip loop only iterates typeCases, which has geomSRID0 but no non-zero SRID, so the common path isn't exercised either. Adding a srid:4326 case to typeCases covers the happy path and would have caught this — worth pairing the fix with that test case.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good catch, will update the tests to include this.

Comment thread table/internal/geo_codec.go Outdated
// specific language governing permissions and limitations
// under the License.

package internal

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is package internal, and WKTToWKB's only callers are _test.go files, but go-geom is now a direct require in go.mod — so every consumer of table/internal (scanner, DV writer) transitively pulls it into the production module graph.

I'd move this into a _test.go file in package internal_test (the test already lives there) or a small testutil package, so go-geom stays test-only. Non-blocking from my side, and I haven't seen it raised on the threads — if the maintainers are fine carrying it that's a reasonable call — but I'd lean toward keeping it out of the production deps.

Comment thread table/arrow_utils.go
}

switch {
case checkCRSString(meta.CRS):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A small cleanup cluster in this function:

The if len(meta.CRS) > 0 wrappers inside both this checkCRSString arm and the checkCRSSJSON arm below are dead — the guard at the top of the function already returns on empty, and the check* funcs only return true for non-empty input (staticcheck SA9003). I'd drop both wrappers.

On the write path (line 1984), strings.EqualFold(lowerCRS, "EPSG:4326") runs a case-fold against an already-lowercased string — lowerCRS == "epsg:4326" is enough.

And in the JSON-object code parsing (line 1923), || codeNum.String() == "" is unreachable — a successful json.Number unmarshal is never empty. I'd drop just that clause; the trailing if code == "" is reachable for {"code":""}, so leave that one.

Comment thread table/internal/geo_codec.go Outdated
// WKTToWKB is a helper which converts Well Known Text (WKT) to Well Known Bytes (WKB).
// Note that return bytes are little endian.
func WKTToWKB(s string) (geoarrow.WKBBytes, error) {
geometry, err := wkt.Unmarshal(s)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

geo_codec_test.go is all happy-path — no invalid-WKT case. I'd add a wantErr row (e.g. "not wkt") so the error path is exercised. While we're here, the wkt.Unmarshal / wkb.Marshal errors return bare; wrapping them (fmt.Errorf("parse WKT: %w", err)) makes failures legible at the call site.

Comment thread table/arrow_utils.go Outdated
return len(b) > 0 && b[0] == '"'
}

func checkCRSSJSON(rawCrs json.RawMessage) bool {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: checkCRSSJSON has a double S — rename to checkCRSJSON at the def and the call site.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch!

Signed-off-by: happydave1 <dzhao2004@gmail.com>
Signed-off-by: happydave1 <dzhao2004@gmail.com>
Signed-off-by: happydave1 <dzhao2004@gmail.com>
Signed-off-by: happydave1 <dzhao2004@gmail.com>
Signed-off-by: happydave1 <dzhao2004@gmail.com>
Signed-off-by: happydave1 <dzhao2004@gmail.com>
Signed-off-by: happydave1 <dzhao2004@gmail.com>
Signed-off-by: happydave1 <dzhao2004@gmail.com>
Signed-off-by: happydave1 <dzhao2004@gmail.com>
@happydave1

Copy link
Copy Markdown
Contributor Author

Thanks for the comments and review guys. I went through and should have addressed all comments for this round.

  1. Fixed the lowercase slicing bug in icebergCRSToGeoArrowMetadata by slicing the original crs instead of lowerCRS
  2. Moved WKTToWKB to geo_codec_test.go to remove go-geom as a production dependency
  3. Added an error case to TestWKTToWKB, also adding error wrapping in wktToWKB
  4. cleaned up dead arms in geoArrowCRSToIcebergCRS

Thank you again for the catches and let me know if I missed anything. Thanks!

@paleolimbot paleolimbot left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just nits from my end. Thank you!

Comment thread table/arrow_utils.go
Comment on lines +686 to +687
// Always add an edge to differentiate between Geography and Geometry arrow fields.
// Note that the edge convention is a best-effort hint and planar geography from other clients won't round-trip through Arrow alone.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this comment is misleading (planar geography is not a thing, and this is not a hint, it affects correctness).

Suggested change
// Always add an edge to differentiate between Geography and Geometry arrow fields.
// Note that the edge convention is a best-effort hint and planar geography from other clients won't round-trip through Arrow alone.
// Always add an edge to differentiate between Geography and Geometry arrow fields.

Comment thread table/arrow_utils_test.go Outdated
Signed-off-by: happydave1 <dzhao2004@gmail.com>

@laskoviymishka laskoviymishka left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's in a good shape now, lets merge it! :shipit:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(table): WKB encoding + GeoArrow conversion for geometry/geography

5 participants